The present disclosure relates to an image processing apparatus, an image processing method, and a program, and more particularly, to an image processing apparatus, an image processing method, and a program for generating a multi-view point image which is applied to three-dimensional (3D) image display.
A naked eye-type 3D display apparatus has been put into practice. The naked eye-type 3D display apparatus allows a user to perceive a stereoscopic image without wearing glasses in three-dimensional (3D) image display processing. The naked eye 3D display apparatus includes, for example, a lenticular sheet or a parallax barrier (parallax barrier) on a display surface, which controls images entering into the left eye and the right eye in accordance with a viewing/listening position.
With such method, however, a correct stereoscopic vision can be obtained only at a limited viewing/listening position with respect to a display. Therefore, when a user's observation position is located at a position different from a specified position, reversed vision and crosstalk occurs. In the reversed vision, a right eye image enters into the left eye, and a left eye image enters into the right eye. In the crosstalk, a left eye image and a right eye image are mixed.
In order to solve this problem, a configuration has been suggested to generate and display not only a standard left eye image and a standard right eye image corresponding to a regular observation position but also an image from a new view point which is configured not to produce any crosstalk when observed from other observation positions.
Not only an original set of a left eye image and a right eye image but also images of other virtual view points are generated as multi-view point images, and an appropriate set of a left eye image and a right eye image according to a user's observation position is selectable from these multi-view point images, in accordance with the observation position, whereby images are displayed while reducing the reversed vision and the crosstalk.
In other words, this allows a user to observe a different pair of a left eye image and a right eye image in accordance with the user's observation position, so that even when the user's observation position is changed, this allows the left eye and the right eye of the observer to observe a left eye image and a right eye image according to each observation position.
More specifically, based on the original images for two view points which are input to a display apparatus or an image processing apparatus, i.e., two view point images including a left eye image (L image) and a right eye image (R image) for 3D image display, view point images for virtual view points are generated in addition to these two view points. For example, multi-view point images for ten different view points including the original LR images are generated.
A user observes a combination of appropriate two images among the generated multi-view point images in accordance with a user's observation position with respect to the display, whereby 3D images can be displayed and can be observed while reducing crosstalk in which a left eye image and a right eye image are mixed, at various observation positions.
For example, Japanese Patent Application Laid-Open No. 2006-115198 discloses a method for inputting an original left eye image (L image) and an original right eye image (R image), executing parallax detection from these two images, and generating images for multiple virtual view points, on the basis of the detected parallax information. More specifically, parallax is detected from the two received original 3D images including the left eye image (L image) and the right eye image (R image), and determining virtual view point positions different from the received LR images, on the basis of the amount of crosstalk and the fusional parallax range.
In the processing described in this Japanese Patent Application Laid-Open No. 2006-115198, however, the quality of the generated virtual view point images is not taken into consideration, and the processing described in this Japanese Patent Application Laid-Open No. 2006-115198 is configured to determine the virtual view point positions using the center of the left eye image and the right eye image as a reference. Therefore, the quality of the generated virtual view point images is reduced, and an image which can be hardly observed may be displayed.
There is a close relationship between the virtual view point position and the image quality.
For example, where the view point position of the received L image is 0.0, and the view point position of the received R image is 1.0, the relationship between an image for a newly generated virtual view point and its image quality has the following features.
(Feature 1) At a virtual view point position between 0.0 and 1.0, i.e., between the L image (0.0) and the R image (1.0), a virtual view point image of 0.5 which is the central position of the LR images has the lowest image quality as compared with other virtual view point positions.
(Feature 2) At a virtual view point position which is equal to or less than 0.0 and equal to or more than 1.0 at the left side of the L image or at the right side of the R image, the farther the position is away from the L image or the R image, the lower the quality of video becomes.
Such relationship between the virtual view point position and the image quality results from, for example, the precision of the parallax detection and the amount of occlusion region included in the image.
It should be noted that when the view point position of 0.0, the original received left eye image can be used as it is, and at the view point position of 1.0, the original received right eye image can be used as it is. Therefore, at these positions, the image quality becomes the highest.
Japanese Patent Application Laid-Open No. 2006-115198 suggests a method for detecting the maximum amount of parallax from 3D images of an original received left eye image (L image) and an original received right eye image (R image) and determining virtual view point positions so that the maximum parallax is accommodated within the amount of crosstalk and the fusional parallax range. In other words, this discloses a method for determining a view point interval of virtual view point image generated according to the maximum amount of parallax of the received LR images.
However, when the maximum parallax is detected from the original LR images, an image in an image region having the maximum parallax and likelihood an image region attracts attention are not taken into consideration. Therefore, for example, the following problems occur.                During the maximum parallax detection, the size of area of the image region having the maximum parallax is not taken into consideration. Therefore, when an object having a small size of area has the maximum parallax, the virtual view point interval may be reduced more than necessary, in accordance with the existence of the maximum parallax image region that hardly affects the visual appearance.        In addition, the likelihood of getting attraction to the image region having the maximum parallax is not taken into consideration during the maximum parallax detection. Therefore, when an image region that hardly attracts attention in terms of visual appearance has the maximum parallax, the virtual view point interval may be reduced or increased more than necessary, in accordance with the maximum parallax information of the image region that hardly affects the visual appearance.        
Japanese Patent Application Laid-Open No. H9-121370 discloses a method using an original received left eye image (L image) and an original received right eye image (R image) to maintain parallax within a fusional range by moving these images in parallel (shifting).
By generating virtual view point images by means of shift processing disclosed in Japanese Patent Application Laid-Open No. H9-121370, the offset of the parallax distribution can be adjusted, i.e., the offset adjustment can be done to move an too-deep image to the viewer's side as a whole. However, since the extension of the parallax distribution may not be adjusted, the following glitches may occur: the entire image may shift too much to the viewer's side or may move too much to the deeper side as a result of the offset adjustment.